The present disclosure relates to image processors, image processing methods, and digital cameras suitable for performing image recognition and feedback operation at high speed with low power consumption in image generation and processing of reduced-size images obtained from an original image with a single-sensor color imaging device.
In the field of recent single-lens reflex digital cameras equipped with large-size imaging devices, single-lens reflex digital cameras requiring neither optical viewfinders nor movable mirrors have attracted attention. A mirror-less single-lens reflex digital camera enables the user to determine a composition of a picture while checking an image of a subject on a liquid crystal display (LCD) monitor, and to take still or video images through automatic focus control and automatic exposure control by using output data from an imaging device. This feature is typically employed in widely used digital cameras and cameras installed in cellular phones.
As described above, a typical digital camera is equipped with a dedicated LCD monitor and can be used when the user determines a composition of a picture while checking an image of a subject on an LCD monitor. At the same time, the camera automatically adjusts exposure and focus in a specified area through face recognition and motion recognition so as to capture still or video images.
An imaging system typified by such a digital camera includes a color separation filter on an imaging device in order to capture color images. A color separation filter known as a Bayer filter, for example, includes primary color filters, i.e., red (R), green (G), and blue (B) color filters, arranged in a square grid, corresponding to pixels of the imaging device. Specifically, in the Bayer pattern, filters of the same color component are placed at every other pixel position along both the sensor-reading and vertical directions of the pixel array. Image data obtained through the color separation filter and serving as an output of the imaging device needs to be handled in a manner that maintains the Bayer pattern in pre-input processing in order to reproduce the colors of the subject.
A small-size image to be displayed is generated from a large-size original image output from an imaging device and is displayed in the following manner. First, the synchronization process of obtaining information on R, G, and B for each pixel is performed with dynamic-range adjustment being performed or white balance being obtained based on an original image. Thereafter, the resulting data is subjected to color image processing to be converted into YCrCb data, which is typically handled. The YCrCb data is then subjected to a reduction (resizing) process to conform to a display size on an LCD monitor. Finally, the resulting data is subjected to a display process to be adjusted in accordance with monitor characteristics. In this manner, the resulting data is displayed on a dedicated monitor.
FIG. 9 is a conceptual view illustrating an internal signal processing block of a typical digital camera.
In FIG. 9, an optical image of a subject that has passed through an optical lens 11 is formed on an imaging device plane, and is output from an imaging device. RAW image data 101 having a Bayer pattern, which is an original image (one surface) of a single-sensor imaging device, is subjected to processes such as DC level adjustment and gain adjustment in a pretreatment section 102, is temporarily written in a memory section 108 via a memory control section 107, is read out from the memory section 108 via the memory control section 107 in a next process, and then is input to an image signal process and reduction (resizing) processor 109. The image signal process and reduction (resizing) processor 109 performs an image signal process of converting a RAW image having a Bayer pattern into a YCrCb data image. The image signal process and reduction (resizing) processor 109 converts the RAW image into the YCrCb data image, and in order to reduce the image to an image size for display, performs a reduction (resizing) process to generate an image of a desired display size. The generated YCrCb data image of a display size is written in the memory section 108 again via the memory control section 107.
At the same time, after the conversion into the YCrCb data image, in order to reduce the image to a size for face detection, the image signal process and reduction (resizing) processor 109 performs a reduction (resizing) process on a luminance signal Y to generate a desired luminance image data for face detection. The luminance image data for face detection from the luminance signal Y is written in the memory section 108 via the memory control section 107, and is suitably read out again from the memory section 108 via the memory control section 107 by a face detection processor 106, and face information such as the position and size of a face in the image is extracted. The extracted information is written in the memory section 108 via the memory control section 107. The face detection information written in the memory section 108 is read out by a CPU 114 via the memory control section 107, is converted into display information indicating a face position. In this manner, display data is generated and written in the memory section 108. At the same time, the CPU 114 automatically adjusts exposure and focus for a specific area based on face recognition information.
The YCrCb data image of a display size written in the memory section 108 is read out from the memory section 108 via the memory control section 107 again, and is input to a display processor 117. At this time, the display information indicating the face position is also read out from the memory section 108, and is input to the display processor 117.
The YCrCb data image and the display information indicating the face position that have been input to the display processor 117 are converted to conform to monitor characteristics of a monitor 118 in the display processor 117, and are output to the monitor 118 to be displayed thereon.
Similarly, in the case of recoding a video image of a standardized size, the original image 101 is converted into a YCrCb data image in the image signal process and reduction (resizing) processor 109, and then is subjected to a reduction (resizing) process in the image signal process and reduction (resizing) processor 109, thereby generating an image of a desired video recording size. The generated YCrCb data image is written in the memory section 108 again via the memory control section 107. The YCrCb data image of a video recording size written in the memory section 108 is read out from the memory section 108 via the memory control section 107 again, and is input to a compressor/decompressor 110. The YCrCb data image input to the compressor/decompressor 110 is subjected to data compression using a video image codec method such as MJPEG, MPEG, or H264, and is written in the memory section 108 via the memory control section 107 again. The compressed video data is read out from the memory section 108 via the memory control section 107 again, and is written on a recording medium 112 through a recording medium interface 111.
With respect to the internal signal processing of the digital camera, FIG. 10 illustrates a series of pipeline processes starting from an input of RAW image data.
In FIG. 10, in a first frame period in which RAW image data 101 having a Bayer pattern is input, the pretreatment section 102 performs DC level adjustment, gain adjustment, etc., and results of these processes are temporarily written in the memory section 108 via the memory control section 107. Then, in processing in a second frame period, the RAW image data is read out via the memory control section 107, is input to the image signal process and reduction (resizing) processor 109, and is subjected to an image signal process to be converted into an YCrCb data image.
In the second frame period, the image signal process and reduction (resizing) processor 109 performs a reduction (resizing) process, and generates an image of a desired display size and luminance image data for face detection. The generated YCrCb data image of the display size and luminance image data for face detection are written in the memory section 108 again via the memory control section 107.
Thereafter, in a third frame period, the luminance image data for face detection is suitably read out from the memory section 108 via the memory control section 107 again by the face detection processor 106 via the memory control section 107, and face information such as the position and size of a face in the image is extracted. The extracted information is written in the memory section 108 via the memory control section 107.
Subsequently, in a fourth frame period, the extracted face information written in the memory section 108 is read out by the CPU 114 via the memory control section 107, and the extracted face information indicating the face position is converted into display information data, which is then written in the memory section 108. At the same time, the CPU 114 starts feedback control for automatically adjusting exposure, focus, and image quality for a specific area based on the face recognition information.
As described above, the image processing illustrated in FIGS. 9 and 10 is performed in the following manner. To determine a composition of a picture while checking an image of a subject on an LCD monitor for image capturing, the process of temporarily storing a single type of reduced-size RAW data with an RGB Bayer pattern in a memory, subjecting the data to a color development image process of converting the data into YCrCb data, and then storing the resulting data in the memory again, includes signal processing in which the RAW data is subjected to a reduction (resizing) process to be reduced to the sizes suitable for video recording, face detection, and a display monitor, is written in a memory section, and then is read out from the memory section again to be subjected to a video image compression process, a face detection process, and a display process.
Typically, to confirm the reproducibility of details of a subject, it is necessary to use a large-size image. On the other hand, to confirm a composition of a picture of a subject, brightness, tones of colors, the presence of a halation, etc. on an LCD monitor or to recognize the position and size of a face through face detection when shooting a picture, it is unnecessary to use such a large-size image, and a small-size image is sufficient for the confirmation and the recognition instead. At present, the number of pixels in an imaging device is increasing and the high-speed read technique is advancing. Under these circumstances, when an image is captured in a record mode called small RAW in which the image is smaller than its original image or when a video image having a standardized size, such as an HD video image, is captured, the size of a RAW image is reduced before processing such as a display process, a face recognition process, and a recording process is performed. This size reduction can remove a redundant process that would otherwise be performed in a subsequent process. As a result, the processing time can be significantly reduced, and operation with low power consumption of the system can be achieved.
In existing techniques of reducing (resizing) a RAW image of RGB with a pretreatment, as presented in, for example, Japanese Unexamined Patent Publications Nos. 2003-346143, 2001-245141, and 2002-84547, color separation into individual pieces of R, G, B data is performed or pixels of the same color are mixed in an initial process for decimation in reducing (resizing) RAW data of RGB as an original image.